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Abstract 

The master equation and, more generally, Markov processes are routinely used as models for 
stochastic processes. They are often justified on the basis of randomization and coarse-graining 
assumptions. Here instead, we derive n'^-order Markov processes and the master equation as 
unique solutions to an inverse problem. In particular, we find that when the constraints are not 
enough to uniquely determine the stochastic model, the n th -ovdex Markov process emerges as 
the unique maximum entropy solution to this otherwise under-determined problem. This gives a 
rigorous alternative for justifying such models while providing a systematic recipe for generalizing 
widely accepted stochastic models usually assumed to follow from first principles. 
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I. INTRODUCTION 



Markov chains are often the starting point for modeling condensed phase stochastic 

dynamics in biophysics (sUs] and beyond jsf. Markov chains are approximations of contin- 
uous system dynamics. They are often justified on the basis of coarse-graining approxima- 



tions 



10|. Coarse-graining reduces classical phase space -with phase points dynamics gov- 



erned by Liouville's equations- to a discrete set of states -with stochastic hopping between 
states determined by stationary transition probabilities. Such coarse-graining methods have 
recently been used to show how Markov models can describe the continuous dynamics of 
biomolecules evolving in complex potential landscapes llMl3|. 



A very different approach to stochastic dynamics is due to Filyukov and Karpov [l4( and 
later Jaynes 15] . Using this approach, models for a stochastic dynamics can be inferred 
as unique solutions to an inverse problem. To be clear, by model we mean the probability 
distribution for the stochastic paths the system can take. 



Normally, the number of stochastic paths greatly outnumbers the constraints imposed. 
To find a unique solution to this under-determined problem we ask: which model is it that 
not only satisfies the limited experimental constraints but also maximizes the entropy for 
the path probabilities? As Shore and Johnson [16j showed, this is exactly equivalent to 
finding a model for the path probabilities which satisfies the experimental constraints while 
satisfying these logical consistency axioms: 1) when A and B are independent data then the 
model for P(A and B) must reduce to P(A)P(B) and the model for P(A or B) must reduce 
to P(A) + P(B); 2) furthermore, any prediction made from the model must be independent 
of the coordinate system used in the calculation. 



ar to the maximum en- 
. In earlier work, 
2211— showed that 
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This method of finding a stochastic model is mathematically simi 
tropy principle for determining equilibrium probability distributions 
Ge et al. -which extended the work of Stock et al. 21[ and Ghosh < 
the 1 st order Markov chain emerges as a natural consequence of path entropy maximization. 
Here we generalize this work in many important ways. 1) We do not limit ourselves to first 
order Markov processes; 2) we consider under which conditions the master equation emerges 
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as a solution to the procedure of path entropy maximization; 3) we consider how different 
types of constraints affect the emergent model; 4) we consider very general (non-linear) 
constraints. 



To the best of our knowledge, this is the first time the master equation and, more gen- 
erally, n th -order Markov processes are rigorously shown to follow from maximum entropy 
principles. This provides an alternative justification for the master equation -the basic tool 
of stochastic physics and biology- which is distinct from standard chemical or mechanis- 
tic justifications provided by van Kampen 1], Zwanzig 23], Gillespie 24j and others. 
The master equation assumes from the onset a dynamics described by stationary transition 
probabilities and time-varying state occupation probabilities. Here we only assume data 
of a specific type is available and the basic logical consistency axioms required to justify 



maximum entropy as an inference tool 



161 ] . Posing the master equation as the solution of 



an inverse problem is significant because possible generalizations to the master equation are 
now derivable within this formalism. These generalizations can then be justified on the firm 
axiomatic basis of provided by Shore and Johnson. 



II. MARKOV MODEL OF n th ORDER: DEFINITIONS AND NOTATIONS 



In this section, we briefly introduce the mathematical notation necessary for the re- 
mainder of the paper. Now, consider a stochastic process in discrete time. Let the index 
it denote the state of the system at time t along the path C from time to T where 
C = {io, h, %2i ■ • ■ ir}- The probability distribution of path C is 



P(C)=p(i ,i 1 ,---i T ) (1) 



An n-point joint probability is defined as follows 



p(ax, ■ ■ ■ a n ; t) = p(i , i u ■ ■ ■ i t _ n , oi, • • • a n , ji, j 2 , ■ ■ -jr-t) ( 2 ) 

io,h,— h—n.,jl,32,— 3T-t 
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where, for sake of generality, ai,---a n are neither the first nor the last t indices of 
p(io,ii, • • • it)- The explicit time index is required, as the result depends on which in- 
dices are summed over. Conditional -also called transition- probabilities are obtained by 
invoking Bayes' theorem: 

p(i , ■ ■ ■ i t _i -> i t ) = p{i , ■ ■ ■ i t _i \i t ) = — (3) 

P{io, ■ ■•it-i) 

We call p(io, • • • it-i ~ > h) a transition probability. When the transition probability depends 
only on the previous n-time steps 

P[lO, ■ ■ ■ lt-1 ->■ It) = P{lt-n, lt-n+1 ■ ■ ■ lt-1 -> h; t) = — : — , (4) 

p(it- n ,---it-i;t) 

the process is called an n th -order Markov process. When the transition probability is time- 
independent, it is called a time-homogeneous Markov process. When no specification is given, 
a Markov process is assumed l s *-order, time- homogeneous. 



III. DERIVATION OF FIRST ORDER MARKOV PROCESS WITH LINEAR 
CONSTRAINTS 

Here we show how the first order Markov process is derived from path entropy maximiza- 
tion. We begin with the definition of path entropy 

H = - E P^ojii, • • -ir) logp(i ,«i, • • -ir)- (5) 

{io,h,---ir} 

We consider Ni ands N 2 linear constraints on one and two-point probabilities, respectively: 
F o a) -EE 4 a) P(H5 t)-(T+ 1)E™ = (a = l,..-N 1 ) 

t=0 it 

F i" ] = E E J nl + A^ t)p(h H + i, t) - TJW = 0. ( 7 = 1, • • • N 2 ) (6) 

t=0 ith+i 



and a normalization condition 



p(*o,*i,---*t) = 1- (7) 

{io,ii,—*r} 

These constraints are imposed using Lagrange multiplier. That is, the Lagrange multiplier 
terms are added to the path entropy as follows: 



- E K 4 'o J *ir--^)logp(io,iir"'»V)-^^«(^^e^V(»t;*)-(^ + l)-S| 

{io,U,---it} a=1 \t=0 «t 



(«) 




N 2 /T-l \ / 

+ ZX E E 4tfxP(<t.<m;* + l)-rJ^ +(P + 1) E p(Mi,--*r)-i$) 

7=1 Vt=0 i t i t+ i / \{*0>*1i-»t} 



Extremizing Eq. (|Sj) with respect to p(io, «i, • • • ir), we obtain 

T T-l 

- logp(z , ix, - ■ ■ ir) - E ^ E e £° + E "7 E + P = o- (9) 

a t=0 7 i=Q 

The Lagrange multipliers introduced in Eq. ([8]) are determined by additional equations which 
come from taking the variation of Eq. (|SJ) with respect to these Lagrange multipliers. The 
solution to Eq. is expressed in terms of the Lagrange multipliers as follows 

(T r-i 
p - E E 4°° + E v i E 4t +1 
a t=0 7 t=0 

= exp(p)w(z )G(iio, k)G(i u i 2 ) ■ ■ ■ G(i T _i,i T )v(i T ) (10) 

where the elements of the vector v, v(i), and the elements of the transfer matrix G, G(i,j), 
are defined as follows 

v(i)=exp (-^fe^A 
G(i,j) = exp (-J2^e\ a) /2 + '£u y J$ ) - £/3 a eJ a> /2 j ■ ( n ) 

\ a 7 a / 
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The m-point joint probability distribution, Eq. (J2J), is obtained from Eq. f lTUj) by summing 
over indices it- m , h+i, • • ■ %t as follows 

p(ai, ■ ■ -a m ;t) = ^ p(k,h, • • -it-m, ai, ■ ■ • , a m ,i t+ i, ■ ■ ■ ,ir) 

= exp(p)[v t G^ m+1 ](ai)G(ai, a 2 )G(a 2 , a 3 ) • • • G(a m _i, a m )[G T ~'v](a m ) 
[vtG f - m+1 ](a 1 )G(a 1 ,a 2 )G(a 2 ,a3)---G(a m _ 1 ,a m )[G T - t v](a m ) 

vtG^v • {U) 

Therefore combining Eq. (jlj) and Eq. (1121) . we have 

exp(p)[v t G*- m ](ai)G(ai,a 2 ) • • ■ G(a m , a m+ i)[G T "*v](a m+1 ) 



p(at,---a m -> a m+1 ;t) 



exp(p)[vtG*- m ](ai)G(ai,a 2 )---G(a m „i,a m )[G T -*+ 1 v](a m ) 
G(a m ,a m+ i)[G T_i v](a m+ i) 



[G^+VKa, 



p(a m -)> a m+ i;t). (13) 



The above is indeed a I s * order Markov process though the transition probability has explicit 
time dependence. 



The 1 st order Markov property was also derived in Ref. 25| for the special case of con 



straining one-point and two-point statistics which we now define. One-particle statistics, 
Fq™^ coincide with 



e\ a) = S ita (« = !,-• -iV) (14) 



where the index a of the constraint now goes over each state of the system, N being their 
total number of such states. This constraint simply counts the number of times state a 
is visited over the course of the trajectory (i.e. this constraint is 1 when state index i is 
identical to a). Likewise, two-point statistics corresponds to imposing these i^'^'s 

4y ] = 5 itT 6 j>a (r,a = l,---N) (15) 

where we labelled the constraint by double indices (r, a) instead of the single index 7 for 
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notational convenience. This again simply counts the number of transitions from state r to 
a over the course of the trajectory. 



IV. DERIVATION OF THE TIME-HOMOGENEOUS MASTER EQUATION 

Recall that a master equation requires time dependent state probabilities and time- 
independent transition probabilities. Under what conditions are such approximations valid? 



To answer this question we apply the Perron- Frobenius theorem [26N33| to the G transfer 
matrix of the previous section-a square matrix which by construction is of size N x N and 
has positive elements. According to the theorem, G satisfies the following properties: 

(1) It has a positive real eigenvalue r, called the Perron- Frobenius eigenvalue, such that 
any other eigenvalue A is strictly smaller than r in absolute value, |A| < r. 

(2) There is a left eigenvector yt = (yi,- --Vn) f° r r with positive components. That is, 
y^G = ry^ and > for all i. Similarly, there is a right eigenvector z with positive 
components, such that Gz = rz and z% > for all i. 

(3) Left and right eigenvectors with eigenvalue r are non-degenerate. 

(4) Hindoo S£ = Z y f 

Now re-consider Eq. (fT3|) where 



/ . ,x G(a m , a m+ i)[G T *v](a m+ i) 
p(a m -+ a m+1 ; t) = [Gr -t + i v](flm) " ( 16 ) 



Since the vector v has only non-negative elements, both G T v/r T and v"l"G T /r T have a 
well-defined non-zero limit for T — > oo, 



G T v vtG T 
lim — — = z(y T v); lim — — = (v T z)y T . (17) 
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Therefore, taking the limit T — t — > oo of Eq. f lTS]) and using Eq. f lTT|) . we find 



That is, the transition probability is time-independent in this limit. However, from Eq. (fl2l) . 
the m-point joint probabilities are still explicitly time-dependent when T — t is large 



[ v t G t rn+^ ai )G(a 1 ,a 2 )G(a2,a 3 )---G(a m ^i,a m )z(a m j 
pyai, • • • , a m ; tj — —, (tyj 



and, in particular, this is true for the one-point occupation probability 



r'vTz 



Thus maximizing the path entropy under the linear constraint Eq. up to two-point 
probabilities, which are imposed for infinite duration into the future (T — t — > oo), we obtain 
a time- homogeneous Markov process which is described by 1) time- independent transition 
probabilities and 2) time-dependent one-point occupation probabilities. The resulting evo- 
lution equation for this time-homogeneous Markov process 



p(a;t + l) = Y J Vip;t)p{b ^ a) (21) 

b 

is the celebrated master equation. 

Note the asymmetry in time: the transition probability as well as the joint probabilies 
are time dependent when the limit of t — > oo is taken but T — t is kept finite. This is simply 
due to the fact that the transition probability p(b — > a) is defined in a time-asymmetric 
manner. 
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The last limit to consider is the stationary case, when both T — t and t are large. Then 
the m-point joint probability of Eq. (112p reduces to 



■ a r . 



y(ai)G(ai, a 2 )G(a 2 , a 3 ) ■ ■ ■ G(a m -i, a m )z(a m ) 



(22) 



which is independent of time as are the state occupation probability or any conditional 
probability derived from Eq. ( 1221) . This is to be expected, since we have the time translation 
invariance in the stationary limit, and the same symmetry should appear in the probabilities 
in the absence of additional information. 



Stationarity also trivially follows when the constraints themselves are stationary, which 
are much stronger conditions than those in Eq. (JBjE. Eq. (122]) was also derived in the large 
T limit with (m = T) in Ref. 34j though the stationary Markov process was assumed from 
the onset therein. Likewise, the I s * order Markov process was derived in Ref. 25[ from 
path entropy maximization for the special case of pair statistics constraints, but neither 
conditions for the time-homogeneous process nor stationarity were discussed^]. 



V. TIME-HOMOGENEOUS MARKOV PROCESSES WITH AN ARBITRARY 
INITIAL CONDITION 



Often data comes not only in the form of state occupation probabilities (e.g. how long 
during the course of a single molecule fluorescence experiment did a protein dwell in its 
compact isoform) or transition probabilities. Data may also be available in the form of 
conditions at different points in time (e.g. the sample is pumped into a photoexcited state 
at time t = 0). Are our conclusions on time-homogeneity from the previous section robust 
1 The stationary process is also obtained when the constraint are imposed at each point in time: 

Ft\t) = eifp^t) - =0 (a = 1, • • ■ N,), (t = 0, • ■ -T) 

Ft () (t)= E J^ + A i t-Mn^k + i;t)-j!f () = o. ( 7 = l,---N 2 )(t = 0,---T-l) (23) 



HH+l 

Our result shows that the weaker constraint Eq. ^ can achieve this 
2 Adapted to our notation, it is stated underneath of Eq. (11) of Ref. 



so 



long as < t, T - t. 



25], that p(a, b) oc G(a, b), implying 



that p(a, b) is time-independent. However, since p(a,b) = [v G< 1]{a) v fa^ GT ' Vl(fc) from Eq.!®, this is 

9 

only strictly correct when T — t and t are both large. 



to initial, final or other such conditions? In this section, we briefly show when the time- 
homogeneity of transition probability depends on such conditions. 



Consider an arbitrary condition imposed at time r 



p(a-t = t) =7r(o). (24) 



We then add the term E a A(a)(p(a; r) — 7r(a)) with Lagrange multipliers A(a) (a = 1, • • • N) 
to the constrained entropy, Eq. (JH]). As before, setting the variation with respect to 
p(io, h, ■ - ■ «t) to zero yields 



T T-l 

p(i , ii, - • • ir) = exp(p + A(i r ) - /3 ^ e it + v ^ Jhh+i) 

t=o t=o 

= exp(p + \(i T ))v(i )G(i , i 2 ) • • ■ G(i T -i, iT)v{i T ) 

v(i )-K(i T )G(i , i 2 ) • - • G(i T -i, «t)^(*t) 



Ei ...i T v(jo)7r(j T )G(j ,ji)G(ji,j 2 ) ■ ■ ■ G(j T -i,j T )v(j T ) 



(25) 



where in the last line we used the normalization condition Eq. (j7]) to eliminate p and the 
initialization constraint Eq. (|24|) to eliminate A. We now have 



t < t — m + 1 : 

( _ Ea[v t G1(a)7r(a)[G^- m + 1 ](a,q 1 )G(a 1 ,a 2 )---G'(a m _ 1 ,a m )[G T -V](aJ 

m '"' am ' J EJvtG1(6)7r(6)[G^v](6) 
t-m+l<r<t: 

p(ai, ■ ■ -a m ;t) 

[ v tG*- OT + 1 ](ai)G(ai,a2) • • •G ! (a r _ t+m _i,a T _ t+m )7r(a T _t +m ) 



EJvtG1(6)7r(6)[G^v](6) 
xG(a T _ t+?n , a T -t+m+i) • • ■ C(a m _i, a m )[G T *v](a m ) 

i < r : 

, _ Eq[v t G < - m+1 ](ai)gK,Q2)---g(a m -i,Q m ) 

m '"' am ' J EJvtG1(6)7r(6)[G^v](6) 

x[G T - i ](a m ,a) 7 r(a)[G T ^v](a). (26) 
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Using the definition of the transition probability from Eq. flU) we find 



t <t: 



p(ai, ■ ■ -a m -> a m+ i,t) 



G(a m ,a m+ i)[G T *v](a m+1 ) 
[G T -t+H](a m ) 



t > t : 



p(ai, • • -a m -> a m+ i;t) 



G(a m ,a m+ i) ]>] a [G T '](flm+i, a)7r(a)[G T r 
E 6 [G-^](a OT ,6)7r(6)[G^v](fo) 



v](o) 



(27) 



We notice that the indices ai, • • • a m _i have dropped out from the right hand side of Eq. (l2"7j) . 
We can therefore write 



showing that, once more, we have a I s * order Markov process. Furthermore, the transition 
probability for t > r has exactly the same form as Eq. ffl3l) . independent of the intial 
condition ix. It is therefore time-homogeneous under the limit of large T — t. The same is not 
true of t < r, where the transition probability always depends on the specified condition and 
time- homogeneity requires both large T — t and r — t. As noted earlier, this time-asymmetry 
is a natural consequence of the fact that the definition of the transition probability itself is 
time- asymmetric. 

VI. GENERAL DERIVATION OF n'^-ORDER MARKOV PROCESS FROM PATH 
ENTROPY MAXIMIZATION 

In this section we generalize the arguments of the previous section in two important ways: 
1) we consider constraints on the data up to n + 1-point probabilities 



p(ai, ■ ■ ■ a m -)> a m+ i] t) = p(a m -> a m+1 ; t), 



(28) 



FW({p(i; t)}, {p(i -> j; t)}, ■ ■ ■ {p(i , ■ ■ ■ i n -i -)• in! *)}) = 0. 



(29) 
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and 2) we do not assume that the constraints are linear functions of their arguments 
(as was the case for Eq. (jSJ) ). 

Provided constraints are linear -as was the case in Eq. ([6])- most of the arguments in 
the previous sections are generalizable to n t/l -order Markov processes. Indeed, the path 
probability would be described by the multiplication of rank-(n + 1) tensors rather than 
matrices, like Eq. (JT2"|) . The n th -order Markov process would follow immediately though 
the derivation of the time-homogeneity of various transition probabilities would require the 
difficult task of applying an analogue of the Perron- Frobenius theorem for general tensors. 

Since we want to derive the n th -order Markov process for fully general constraints, as given 
by Eq. ( 1291) . we take a different route. We first express the path probability p(zi,2 2 , ■ ■ - ir) 
in terms of the conditional probabilities: 

p(io,ii, ■■■i T ) = p(io; 0)p(i -> h; l)p(i ,ii ->■ ii\ 2) • • -p(io,ii ■ •■%t-\ -> ir]T). (30) 
Substituting this expression into Eq.([5]), we get 

H = - ^2 P(*o,ii, • • -ir) I logj5(i ;0) + ^logp(i , • • -h -» h+i,t + 1) J 

{io,H,-«T} \ * = / 

T-1 

= ~ ^2 p(i;0) log p(i;0) - p(io,ii,---it+i;t+ 1) logp(i , • ■ -it -»> ^+i;tf31) 

i t=0 {io,h,— H+l} 

where, in getting from first to second line, we invoked the relation between joint and marginal 
probabilities; p(i ■ ■ ■ i m ; m) = £) im+1 ... iT p(h, i2,-" *t)- 

Now reconsider the constraints given by Eq. ( 129|) imposed from t) to p(io, ■ ■ -i n -i — > 
i n ; t). We will maximize the entropy, Eq. (13 in two steps: 1) we maximize the entropy with 
repect to {p(io, {k > n) , for given values of {p(«o, with k < n; 2) we then 

vary the entropy over the remaining variables, {p(zo, ■ • • ik] 0} — n )- assumptions, 
constraints on the data only matter in step 2. Furthermore, as we now show, step 1 (the 
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unconstrained maximization) is sufficient to show that the general path probability reduces 
to that of an n th -ordex Markov process. 

In order to perform step 1, we first invoke the equality 

- ^ ft log ft < -^qilogPi (32) 

i i 

for arbitrary probability distributions pi and g, []. It follows from Eq. (132 p that 

3 

< -} y p{i , ■ ■ • ,i m -i -> j;t) logp(z TO _ n , • • • ,z m _i -> j;t) (34) 
Summing both sides of Eq. (I3"l"|) over i , • • -i m _i, we find 

= - ^2 p(*o>'" > v-i,i;*)iogp(i , ---v-i ->-j;*) 

<- P(^o, ■ • • Wi, j;i) logp(^ m _„, • • -i m -i ^ j;t) (35) 

io>*"*m-l)i 

The above sets a bound on the last term of the path entropy, Eq. ( I3~TT) . Therefore, for given 
values of {p{io, ■ ■ ■ ik',t)} with k < n, we see that is maximized for 

p(«o,-'-,Vi->Jii)=p(Vn,--',Wi^j;i) (m>n), (36) 

the system now being described by a n^-order Markov model where the probability p(i; t) 
is determined only by previous n steps of history. 

3 Using the well-known inequality log a; < — 1 + x for x > 0, we see that 

-^2q l \ogq l +^2q l \ogp l = ^ q % log — < ^^(-1 + ~ ) = + = °' ( 33 ) 

® « ''' I i 

proving the inequality Eq. (|32p . This inequality was also invoked in Ref. [14j in a much narrower setting 
(of deriving a order Markov model). 
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Now Eq. fl5B|) for the path probability n^-order Markov process can be substituted 
into the path entropy formula Eq. (13TT) . Step 2 can be carried forward: the resulting 
path entropy can be maximized with respect to the remaining variables p{io] t) , p(io —> 
i\\ t), ■ ■ -p(io, h, ' ' ' in-i in] t) under the constraints Eq. ff29l) . 

In summary, we have just shown that n'^-order Markov processes follow under very gen- 
eral constraints provided by Eq. ( 1291) . Markov models emerge from the entropy maximiza- 
tion method - and these provide immediate and principled generalizations of the ubiquitous 
master equation. 



VII. DISCUSSION 



Markov processes and master equations -the evolution equation describing a I s * order 
time- homogeneous Markov process- are standard stochastic modeling tools invoked across 
disciplines. Such models are usually justified mechanistically by coarse-graining arguments 
or by assuming quick randomization in space of reactants and products (the "well-stirred" 
approximation). Yet it is challenging to ascertain a priori whether any of these conditions 
actually hold. Just like maximum entropy has provided an alternative to ergodic theory for 
the justification of the equilibrium probability distribution [17] . we believe that the path 
entropy techniques of Filyukov and Karpov 14| , and later Jaynes 15] , provide a compelling 



axiomatic basis for the Markov process and the master equation. Here the Markov process 
emerges as a solution to the following inverse problem: given measurable n-point constraints 
on a trajectory, what is the least biased model for a probability distribution? By least 
biased, we mean one that, for instance, does not impose correlations in a model when 
such correlations are not otherwise warranted by the data (technically these are the logical 
consistency axioms of Shore and Johnson). The unique solution to this problem is that 
which maximizes the entropy subject to constraints from the data. 

With this formalism, we justify generalizations of the master equation on rigorous mathe- 
matical grounds. It is tempting to conjecture whether the n* h -order Markov process can lead 
to a time-homogeneous process so long as the constraints are imposed for a time much longer 

14 



than that of one time step. The proof would require an analogue of the Perron-Frobenius 
theorem for general tensors, an interesting subject for further investigation. 
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